Image processing device, image processing method, program, and endoscope device

ABSTRACT

An image processing device includes an input unit which inputs ordinary frames in a state in which an object is irradiated with ordinary light, and a special frame in a state in which the object is irradiated with special light, which are imaged consecutively at a predetermined ratio according to a predetermined frame period; a detection unit which detects motion vectors of the object from a plurality of the ordinary frames with different imaging timing; a motion correction unit which subjects the special frame to motion correction corresponding to the imaging timing of the ordinary frames based on the detected motion vectors; and a compositing unit which subjects the ordinary frames to an image compositing process based on the special frame.

CROSS REFERENCE TO RELATED APPLICATIONS

This application claims the benefit of Japanese Priority PatentApplication JP 2014-048336 filed Mar. 12, 2014, the entire contents ofwhich are incorporated herein by reference.

BACKGROUND

The present disclosure relates to an image processing device, an imageprocessing method, a program, and an endoscope device. In particular,the present disclosure relates to an image processing device, an imageprocessing method, a program, and an endoscope device, each of which iscapable of combining and displaying an ordinary image which is imaged byirradiating a human body with ordinary light such as white light, and aspecial image which is obtained by irradiating the human body with aspecial light and illustrates the position of blood vessels.

In the related art, for example, with the intention of usage in amedical setting, various technologies are proposed in which an ordinaryimage of an organ or the like that is imaged by an endoscope device iscombined with a special image that represents the position of bloodvessels or a lesion such as a tumor, which are difficult to discern inthe ordinary image, and the result is displayed.

For example, imaging an ordinary image and a special image using timedivision is described in Japanese Unexamined Patent ApplicationPublication No. 2007-313171. As another example, performing compositedisplay of the ordinary image and the special image is described inJapanese Unexamined Patent Application Publication No. 2012-24283.

Here, the term “ordinary image” indicates an image which is imaged byirradiating an organ or the like that serves as the object with ordinarylight such as white light. Hereinafter, the ordinary image will also bereferred to as an ordinary frame. The term “special image” indicates animage which is imaged by irradiating the object with special light of apredetermined wavelength different from that of the ordinary light.Hereinafter, the special image will also be referred to as the specialframe. Note that, when imaging the special image, there is a case inwhich a fluorescent agent or the like which reacts to the irradiation ofthe special light is mixed into or applied to the blood vessel (theblood) or the lesion that serves as the object.

SUMMARY

Since combining the ordinary frame and the special frame that are imagedusing time division causes a shift in the imaging timing, when there ishand shaking or the object moves, there is a likelihood that thealignment of the ordinary frame with the special frame may not beperformed accurately.

Note that, technology also exists which carries out the compositingafter detecting motion vectors between the ordinary frame and thespecial frame that are imaged using time division and motion correctionis performed based on the motion vectors. However, the imagingconditions differ between the ordinary frame and the special frame,errors occur easily in block matching when detecting the motion vectors,and it is difficult to accurately detect the motion vectors.

It is desirable to enable the accurate alignment and combination of anordinary frame and a special frame that are imaged using time division.

According to a first embodiment of the present disclosure, there isprovided an image processing device which includes an input unit whichinputs ordinary frames in a state in which an object is irradiated withordinary light, and a special frame in a state in which the object isirradiated with special light, which are imaged consecutively at apredetermined ratio according to a predetermined frame period; adetection unit which detects motion vectors of the object from aplurality of the ordinary frames with different imaging timing; a motioncorrection unit which subjects the special frame to motion correctioncorresponding to the imaging timing of the ordinary frames based on thedetected motion vectors; and a compositing unit which subjects theordinary frames to an image compositing process based on the specialframe.

In the image processing device, the image processing device may furtherinclude a feature extraction process unit which generates a featureextraction frame by subjecting the special frame to a feature extractionprocess, the motion correction unit may further subject the featureextraction frame to motion correction corresponding to the imagingtiming of the ordinary frames based on the detected motion vectors, andthe compositing unit may subject the ordinary frames to the imagecompositing process based on the feature extraction frame that issubjected to motion correction.

In the image processing device, the feature extraction process unit maygenerate a differential filter frame as the feature extraction frame bysubjecting the special frame to a differential filter process.

In the image processing device, the compositing unit may subject theordinary frames to a superposing compositing process or a markingcompositing process as the image compositing process.

In the image processing device, as the superposing compositing process,the compositing unit may add the motion-corrected special frame to theordinary frames according to the motion-corrected feature extractionframe.

In the image processing device, as the marking compositing process, thecompositing unit may subject the ordinary frames to a color conversionprocess according to the motion-corrected feature extraction frame.

In the image processing device, the compositing unit may subject theordinary frames to a superposing compositing process or a markingcompositing process as the image compositing process according to aselection by a user.

In the image processing device, the image processing device may furtherinclude a motion vector correction unit which corrects the detectedmotion vectors based on the plurality of motion vectors that areconsecutively detected.

According to a first embodiment of the present disclosure, there isprovided an image processing method performed by an image processingdevice. The method includes inputting ordinary frames in a state inwhich an object is irradiated with ordinary light, and a special framein a state in which the object is irradiated with special light, whichare imaged consecutively at a predetermined ratio according to apredetermined frame period; detecting motion vectors of the object froma plurality of the ordinary frames with different imaging timing;subjecting the special frame to motion correction corresponding to theimaging timing of the ordinary frames based on the detected motionvectors; and subjecting the ordinary frames to an image compositingprocess based on the special frame.

According to a first embodiment of the present disclosure, there isprovided a program for causing a computer to function as an input unitwhich inputs ordinary frames in a state in which an object is irradiatedwith ordinary light, and a special frame in a state in which the objectis irradiated with special light, which are imaged consecutively at apredetermined ratio according to a predetermined frame period; adetection unit which detects motion vectors of the object from aplurality of the ordinary frames with different imaging timing; a motioncorrection unit which subjects the special frame to motion correctioncorresponding to the imaging timing of the ordinary frames based on thedetected motion vectors; and a compositing unit which subjects theordinary frames to an image compositing process based on the specialframe.

In the first embodiments of the present disclosure, ordinary frames in astate in which an object is irradiated with ordinary light, and aspecial frame in a state in which the object is irradiated with speciallight, which are imaged consecutively at a predetermined ratio accordingto a predetermined frame period are input; motion vectors of the objectfrom a plurality of the ordinary frames with different imaging timingare detected; the special frame is subjected to motion correctioncorresponding to the imaging timing of the ordinary frames based on thedetected motion vectors; and the ordinary frames are subjected to animage compositing process based on the special frame.

According to a second embodiment of the present disclosure, there isprovided an endoscope device which includes a light source unit whichirradiates an object with ordinary light or special light; an imagingunit which consecutively images, at a predetermined ratio according to apredetermined frame period, ordinary frames in a state in which theobject is irradiated with the ordinary light, and a special frame in astate in which the object is irradiated with the special light; adetection unit which detects motion vectors of the object from aplurality of the ordinary frames with different imaging timing; a motioncorrection unit which subjects the special frame to motion correctioncorresponding to the imaging timing of the ordinary frames based on thedetected motion vectors; and a compositing unit which subjects theordinary frames to an image compositing process based on the specialframe.

In the second embodiment of the present disclosure, an object isirradiated with ordinary light or special light; ordinary frames in astate in which the object is irradiated with the ordinary light, and aspecial frame in a state in which the object is irradiated with thespecial light are consecutively imaged at a predetermined ratioaccording to a predetermined frame period; motion vectors of the objectare detected from a plurality of the ordinary frames with differentimaging timing; the special frame is subjected to motion correctioncorresponding to the imaging timing of the ordinary frames based on thedetected motion vectors; and the ordinary frames are subjected to animage compositing process based on the special frame.

According to the first embodiments of the present disclosure, it ispossible to accurately align and combine ordinary frames and a specialframe that are images using time division.

According to the second embodiment of the present disclosure, it ispossible to image ordinary frames and a special frame using timedivision, and to accurately align and combine the frames.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram illustrating a configuration example of anendoscope device to which an embodiment of the present disclosure isapplied;

FIG. 2 is a diagram illustrating imaging timing between ordinary framesand special frames;

FIG. 3 is a block diagram illustrating a detailed configuration exampleof an image processing unit of FIG. 1;

FIG. 4 is a block diagram illustrating a detailed configuration exampleof a motion vector detection unit of FIG. 3;

FIG. 5 is a flowchart illustrating an image compositing process;

FIG. 6 is a diagram illustrating an example of motion correction amountestimation;

FIG. 7 is a diagram illustrating an impression of correcting dispersionin motion vectors based on a series of motion vectors;

FIG. 8 is a diagram illustrating an impression of a superposingcompositing process; and

FIG. 9 is a block diagram illustrating a configuration example of acomputer.

DETAILED DESCRIPTION OF EMBODIMENTS

Hereafter, detailed description will be given of a favorable embodimentfor realizing the present disclosure (referred to below as the“embodiment”) with reference to the drawings.

Configuration Example of Endoscope Device

FIG. 1 illustrates a configuration example of an endoscope device, whichis an embodiment of the present disclosure, that images an ordinaryframe and a special frame using time division, accurately aligns andcombines the frames, and displays a composite frame that is obtained asa result.

An endoscope device 10 is configured to include a light source unit 11,an imaging unit 12, a developing unit 13, an image processing unit 14,and a display unit 15.

The light source unit 11 switches between ordinary light such as whitelight and special light that has a predetermined wavelength for eachframe that is imaged, and irradiates the object (an organ or the like inthe body) therewith. The light source unit 11 outputs an irradiationidentification signal indicating which of the ordinary light and thespecial light the object is irradiated with to the image processing unit14 for each frame that is imaged. Note that, when irradiating the objectwith the special light, an optical filter which transmits only apredetermined wavelength may be provided in the light path of theordinary light.

The imaging unit 12 images the object in a state in which the ordinarylight or the special light is radiated from the light source unit 11,and outputs an image signal that is obtained as a result to thedeveloping unit 13. The developing unit 13 subjects the image signalthat is input thereto from the imaging unit 12 to a developing processsuch as a mosaic process, and outputs the image signal resulting fromthe process (the ordinary frame when the ordinary light is radiated, andthe special frame when the special light is radiated) to the imageprocessing unit 14.

Here, in the special frame, the blood vessel or the lesion such as atumor is made clearer in comparison to in an ordinary frame; however, incontrast, the brightness of the entire frame is dark and there is muchnoise. Meanwhile, in the ordinary frame, the entire frame is bright incomparison to the special frame, and there is little noise; however, incontrast, it is difficult to distinguish the blood vessel or the lesionsuch as a tumor.

The image processing unit 14 detects motion vectors using two ordinaryframes with different imaging timing. By subjecting the special frame toa differential filter process, the image processing unit 14 generates aframe (hereinafter referred to as a differential filter frame) in whichedge portions (specifically, the contours or the like of the bloodvessel or the lesion, for example) within the image are emphasized.Furthermore, the image processing unit 14 performs motion correction oneach of the special frame and the differential filter frame based on themotion vectors that are detected from the ordinary frame, combines theordinary frame with the special frame and the differential filter framewhich are subjected to the motion correction, and outputs the compositeframe that is obtained as a result to the display unit 15.

The display unit 15 displays the composite frame.

Imaging Timing of Ordinary Frame and Special Frame

Next, an example of the imaging timing of the ordinary frames and thespecial frames is illustrated in FIG. 2.

In the endoscope device 10, ordinary frames are imaged for severalcontinuous frames, and a special frame is imaged periodically betweenthe ordinary frames. For example, as illustrated in FIG. 2, the imagingratio of ordinary frames to special frames is set to 4:1.

However, the ratio is not limited to 4:1, and may be variable. In FIG.2, Ta illustrates a timing at which an ordinary frame is imaged oneframe before a special frame is imaged, Tb illustrates a timing at whicha special frame is imaged, and Tc, Td, and Te illustrate timings atwhich ordinary frames are imaged 1, 2, and 3 frames, respectively, afterthe special frame is imaged. Ta to Te will be used in the description ofthe detection of motion vectors described later.

Configuration Example of Image Processing Unit 14

Next, a configuration example of the image processing unit 14 isillustrated in FIG. 3.

The image processing unit 14 is configured to include a switching unit21, a motion vector detection unit 22, a correction amount estimationunit 23, a frame memory 24, a differential filter processing unit 25, amotion correction unit 26, and a compositing unit 27.

In the image processing unit 14, the ordinary frames and the specialframes that are input thereto from the developing unit 13 of theprevious stage are input to the switching unit 21, and the irradiationidentification signal from the light source unit 11 is input to theswitching unit 21, the motion vector detection unit 22, and thecorrection amount estimation unit 23.

The switching unit 21 determines whether or not the input from thedeveloping unit 13 is a special frame based on the irradiationidentification signal, when the input is not a special frame (is anordinary frame), outputs the ordinary frame to the motion vectordetection unit 22 and the compositing unit 27, and when the input is aspecial frame, the special frame is output to the frame memory 24.

For each frame period, the motion vector detection unit 22 detects themotion vectors using two ordinary frames with different imaging timing,and outputs the detected motion vectors to the correction amountestimation unit 23.

The correction amount estimation unit 23 estimates the motion correctionamounts of the special frame and the differential filter frame based onthe motion vectors that are detected by the motion vector detection unit22, and outputs the estimated motion correction amounts to the motioncorrection unit 26. Note that, the correction amount estimation unit 23is capable of correcting motion vectors which may be erroneouslydetected based on the motion vectors that are detected in succession,and is capable of estimating the motion correction amounts based on thecorrected motion vectors.

The frame memory 24 holds the special frame that is input thereto fromthe switching unit 21, and supplies the held special frame to thedifferential filter processing unit 25 and the motion correction unit 26for each frame period. The frame memory 24 updates the held specialframe when the next special frame is input thereto from the switchingunit 21.

The differential filter processing unit 25 generates a featureextraction frame in which features in the image are emphasized bysubjecting the special frame that is supplied thereto from the framememory 24 to a differential filter process (for example, the Sobelfilter process), and outputs the feature extraction frame to the motioncorrection unit 26. Note that, in the case of the differential filterprocess, a differential filter frame in which the edge portions areemphasized is generated as the feature extraction frame. As describedabove, since the frame memory 24 supplies the special frame held thereinfor each frame period, the same special frames are consecutivelysupplied. In this case, the differential filter processing unit 25 mayomit the differential filter process and output the result of theprevious differential filter process to the motion correction unit 26.

Note that, as described above, instead of generating the differentialfilter frame using the differential filter process, for example, aprocess may be executed in which a region in which the variance or thedynamic range in a micro-block (of 3×3 pixels, for example) is greaterthan or equal to a threshold that is extracted, and a feature extractionframe indicating the extraction results is generated. As anotherexample, a process may be executed in which a region in which the signallevels of pixels are within a specific threshold, that is, a region withspecific RGB levels is extracted, and a feature extraction frameindicating the extraction results is generated. As still anotherexample, a closed region (corresponding to a tumor or the like) may besubjected to a contour detection process such as snakes, and a featureextraction frame indicating the results may be generated.

The motion correction unit 26 subjects the special frame from the framememory 24 to the motion correction based on the motion correctionamounts that are input from the correction amount estimation unit 23,subjects the differential filter frame from the differential filterprocessing unit 25 to the motion correction, and outputs the post-motioncorrection special frame and differential filter frame to thecompositing unit 27.

The compositing unit 27 includes a superposing unit 28 and a markingunit 29, using the ordinary frame and the post-motion correction specialframe and differential filter frame as the input, generates a compositeframe by performing a superposing compositing process by the superposingunit 28 or a marking compositing process by the marking unit 29, andoutputs the composite frame to the display unit 15 of the subsequentstage.

Configuration Example of Motion Vector Detection Unit 22

Next, a configuration example of the motion vector detection unit 22 isillustrated in FIG. 4. The motion vector detection unit 22 is configuredto include frame memories 31 and 32, a frame selection unit 33, a blockmatching unit 34, and a vector correction unit 35.

In the motion vector detection unit 22, the ordinary frame that is inputthereto from the switching unit 21 of the previous stage is input to theframe memory 31 and the frame selection unit 33.

For each frame period, the frame memory 31 outputs the ordinary framethat is held therein until that point to the frame memory 32 and theframe selection unit 33, and updates the data held therein until thatpoint with the ordinary frame that is input from the switching unit 21of the previous stage. In the same manner, for each frame period, theframe memory 32 outputs the ordinary frame that is held therein to theframe selection unit 33, and updates the data held therein with theordinary frame that is input from the frame memory 31 of the previousstage.

However, among frame periods, at a timing at which the ordinary frame isnot input to the motion vector detection unit 22, the frame memory 31outputs the ordinary frame that is held until that point to thesubsequent stage, and clears the data that is held until that point.

At the next timing, since the frame memory 31 is not holding any data,the output to the subsequent stage is not performed. The frame memory 32outputs the ordinary frame that is held until that point to thesubsequent stage, and clears the data that is held until that point.

Therefore, two or three ordinary frames with different imaging timingare input to the frame selection unit 33 at the same time.

When two ordinary frames are input to the frame selection unit 33 at thesame time, the two ordinary frames are output to the block matching unit34. When three ordinary frames are input to the frame selection unit 33at the same time, the two ordinary frames that are input from the framememories 31 and 32 are output to the block matching unit 34. The blockmatching unit 34 detects the motion vectors between the two ordinaryframes using a block matching process.

The vector correction unit 35 determines the relationship between thetwo ordinary frames that are used for the motion vectors based on theirradiation identification signal, corrects the detected motion vectorsbased on the relationship, and outputs the motion vectors to thecorrection amount estimation unit 23.

Detailed description will be given of the correction of the motionvectors by the vector correction unit 35. If the output from the framememory 31 is used as a reference, when the reference imaging timing isthe Ta illustrated in FIG. 2, the ordinary frame from the frame memory32 that is one frame prior to the reference, and the reference ordinaryframe from the frame memory 31 are input to the frame selection unit 33,and the motion vectors are detected from the two ordinary frames. Inthis case, the vector correction unit 35 does not perform the motionvector correction.

When the reference imaging timing is the Tb illustrated in FIG. 2, sincethe Tb is the imaging timing of the special frame, the frame memory 31does not perform output. The ordinary frame from the frame memory 32that is one frame prior to the reference, and the ordinary frame fromthe switching unit 21 that is one frame after the reference are input tothe frame selection unit 33, and the motion vectors are detected fromthe two ordinary frames. In that case, since the detected motion vectorsare from between ordinary frames that are two frames separated from eachother, the vector correction unit 35 multiplies each of the vertical andhorizontal components of the detected motion vectors by ½.

When the reference imaging timing is the Tc illustrated in FIG. 2, thereference ordinary frame from the frame memory 31, and the ordinaryframe from the switching unit 21 that is one frame after the referenceare input to the frame selection unit 33, and the motion vectors aredetected from the two ordinary frames. In that case, since thedirections of the detected motion vectors oppose each other, the vectorcorrection unit 35 multiplies each of the vertical and horizontalcomponents of the detected motion vectors by −1.

When the reference imaging timing is the Td illustrated in FIG. 2, theordinary frame from the frame memory 32 that is one frame prior to thereference, the reference ordinary frame from the frame memory 31, andthe ordinary frame from the switching unit 21 that is one frame afterthe reference are input to the frame selection unit 33, and the motionvectors are detected from the two ordinary frames from the framememories 31 and 32. In this case, the vector correction unit 35 does notperform the motion vector correction.

When the reference imaging timing is the Te illustrated in FIG. 2, theordinary frame from the frame memory 32 that is one frame prior to thereference, the reference ordinary frame from the frame memory 31, andthe ordinary frame from the switching unit 21 that is one frame afterthe reference are input to the frame selection unit 33, and the motionvectors are detected from the two ordinary frames from the framememories 31 and 32. In this case, the vector correction unit 35 does notperform the motion vector correction.

The motion vectors that are corrected as described above are output fromthe vector correction unit 35 to the correction amount estimation unit23 of the subsequent state.

Image Compositing Process by Image Processing Unit 14

Next, description will be given of the image compositing process by theimage processing unit 14 with reference to FIG. 5.

FIG. 5 is a flowchart illustrating an image compositing process. Theimage compositing process is executed for each frame period.

In step S1, the switching unit 21 determines whether or not the inputfrom the developing unit 13 is a special frame based on the irradiationidentification signal, and when the input is a special frame, thespecial frame is output to the frame memory 24. Conversely, when it isdetermined that the input is not a special frame (is an ordinary frame),the switching unit 21 outputs the ordinary frame to the motion vectordetection unit 22 and the compositing unit 27.

In step S2, the frame memory 24 supplies the special frame that is helduntil that point to the differential filter processing unit 25 and themotion correction unit 26. Note that, the frame memory 24 updates theheld special frame when the special frame is input thereto from theswitching unit 21.

In step S3, the differential filter processing unit 25 generates adifferential filter frame in which edge portions in the image areemphasized by subjecting the special frame that is supplied thereto fromthe frame memory 24 to a differential filter process (for example, theSobel filter process such as the one illustrated in the followingequation (1)), and outputs the differential filter frame to the motioncorrection unit 26.

Sobel_(Rh)(x,y)=|−R(x−1,y−1)−2R(x−1,y)−R(x−1,y+1)+R(x+1,y−1)+2R(x+1,y)+R(x+1,y+1)|

Sobel_(Rv)(x,y)=|−R(x−1,y−1)−2R(x,y−1)−R(x+1,y−1)+R(x−1,y+1)+2R(x,y+1)+R(x+1,y+1)|

Sobel_(R)(x,y)=Sobel_(Rh)(x,y)+Sobel_(Rv)(x,y)

Sobel_(Gh)(x,y)=|−G(x−1,y−1)−2G(x−1,y)−G(x−1,y+1)+G(x+1,y−1)+2G(x+1,y)+G(x+1,y+1)|

Sobel_(Gv)(x,y)=−G(x−1,y−1)−2G(x,y−1)−G(x+1,y−1)+G(x−1,y+1)+2G(x,y+1)+G(x+1,y+1)|

Sobel_(G)(x,y)=Sobel_(Gh)(x,y)+Sobel_(Gv)(x,y)

Sobel_(Bh)(x,y)=|−B(x−1,y−1)−2B(x−1,y)−B(x−1,y+1)+B(x+1,y−1)+2B(x+1,y)+B(x+1,y+1)|

Sobel_(Bv)(x,y)=|−B(x−1,y−1)−2B(x,y−1)−B(x+1,y−1)+B(x−1,y+1)+2B(x,y+1)+B(x+1,y+1)|

Sobel_(B)(x,y)=Sobel_(Bh)(x,y)+Sobel_(Bv)(x,y)  (1)

Note that, R, G, and B in the equation (1) respectively correspond tolevels in the R, G, and B planes of the special frame.

In step S4, the motion vector detection unit 22 detects the motionvectors using two ordinary frames with different imaging timing, andoutputs the motion vectors to the correction amount estimation unit 23.In step S5, the correction amount estimation unit 3 determines whetheror not the detected motion vectors are less than or equal to apredetermined threshold, and when the detected motion vectors are lessthan or equal to the predetermined threshold, the process proceeds tostep S6 in order to use the motion vectors in the motion correction.Conversely, when the detected motion vectors are greater than thepredetermined threshold, the motion vectors are not used in the motioncorrection. In this case, the image compositing process that correspondsto the present imaging timing ends.

In step S6, the correction amount estimation unit 23 estimates themotion correction amounts of the special frame and the differentialfilter frame based on the motion vectors that are detected by the motionvector detection unit 22, and outputs the estimated motion correctionamounts to the motion correction unit 26. Specifically, for example, themotion correction amounts H_(x) and H_(y) are computed as illustrated inthe following equation (2).

$\begin{matrix}{{H_{x} = {\sum\limits_{t = 1}^{N}V_{x,t}}}{H_{y} = {\sum\limits_{t = 1}^{N}V_{y,t}}}} & (2)\end{matrix}$

In the equation (2), V_(x) and V_(y) are motion vectors that aredetected and corrected, N represents the imaging timing t=N of theordinary frame for which the motion vectors are detected in relation tothe imaging timing t=0 of the special frame for which correction isperformed.

Note that, in the correction amount estimation unit 23, it is alsopossible to correct the dispersion in the motion vectors andsubsequently estimate the motion correction amounts based on the seriesof motion vectors, as described hereinafter, as another motioncorrection amount estimation method.

FIG. 6 is a diagram illustrating a process flow in which the dispersionin the motion vectors is corrected and the motion correction amounts aresubsequently estimated based on the series of motion vectors. FIG. 7illustrates an impression of correcting the dispersion in the motionvectors based on the series of motion vectors.

Specifically, the motion vectors (V′_(x,t) and V′_(y,t)) in relation tothe imaging timing t are estimated as illustrated in the followingequation (3).

V′ _(x,t) =a _(x) t ³ +b _(x) t ² +c _(x) t+d _(x)

V′ _(y,t) =a _(y) t ³ +b _(y) t ² +c _(y) t+d _(y)  (3)

The motion correction amounts H_(x) and H_(y) are computed using thefollowing equation (4) by substituting the motion vectors of theequation (2) with the estimated motion vectors (V′_(x,t) and V′_(y,t)).

$\begin{matrix}{{H_{x} = {\sum\limits_{t = 1}^{N}V_{x,t}^{\prime}}}{H_{y} = {\sum\limits_{t = 1}^{N}V_{y,t}^{\prime}}}} & (4)\end{matrix}$

Note that, the coefficients (a_(x), b_(x), c_(x), and d_(x)) and (a_(y),b_(y), c_(y), and d_(y)) in the equation (3) can be calculated using theleast squares method using the detected motion vectors (V_(x,1) andV_(y,1)), . . . , (V_(x,t) and V_(y,t)).

After the motion correction amounts are estimated as described above,the process proceeds to step S7.

In step S7, the motion correction unit 26 subjects the special framefrom the frame memory 24 to the motion correction based on the motioncorrection amounts that are input from the correction amount estimationunit 23, subjects the differential filter frame from the differentialfilter processing unit 25 to the motion correction, and outputs thepost-motion correction special frame and differential filter frame tothe compositing unit 27. The ordinary frame, the special frame, and thedifferential filter frame that are input to the compositing unit 27become frames in which the object is accurately aligned in relation toeach other.

In step S8, the compositing unit 27 generates a composite frame bysubjecting the ordinary frame and the post-motion correction specialframe and differential filter frame to the superposing compositingprocess or the marking compositing process according to the selectionfrom the user, and outputs the composite frame to the display unit 15 ofthe subsequent stage.

Description will be given of the superposing compositing process. Asillustrated in the following equation (5), in the superposingcompositing process, the result of multiplying the post-motioncorrection differential filter frame and special frame with each otheris added to the ordinary frame.

O _(R)(x,y)=C ₀ ×N _(R)(x,y)+C ₁×Sobel_(R)(x,y)×I _(R)(x,y)

O _(G)(x,y)=C ₀ ×N _(G)(x,y)+C ₁×Sobel_(G)(x,y)×I _(G)(x,y)

O _(B)(x,y)=C ₀ ×N _(B)(x,y)+C ₁×Sobel_(B)(x,y)×I _(B)(x,y)  (5)

In the equation (5), O(x,y) is a pixel value of the composite frame,N(x,y) is a pixel value of the ordinary frame, Sobel(x,y) is a pixelvalue of the post-motion correction differential filter frame, andI(x,y) is a pixel value of the post-motion correction special frame. C₀and C₁ are coefficients that control the degree of superposition and maybe arbitrarily set by the user.

FIG. 8 illustrates an impression of the superposing compositing processdescribed above. In the superposing compositing process, it is possibleto obtain the composite frame in which the special frame and thedifferential filter frame are accurately aligned to the ordinary frame,and the edges of a portion to be focused on (a blood vessel, a lesion,or the like) are emphasized and superposed.

Next, description will be given of the marking compositing process. Asillustrated in the following equation (6), in the marking compositingprocess, the ordinary frame is subjected to pseudo color conversionusing a color matrix process according to color conversion coefficientsC that are multiplied by the pixel values of the differential filterframe.

$\begin{matrix}{\begin{pmatrix}{O_{R}\left( {x,y} \right)} \\{O_{G}\left( {x,y} \right)} \\{O_{B}\left( {x,y} \right)}\end{pmatrix} = {\begin{pmatrix}{C_{1\; R} \times {{Sobel}_{R}\left( {x,y} \right)}} & {C_{1G} \times {{Sobel}_{G}\left( {x,y} \right)}} & {C_{1B} \times {{Sobel}_{B}\left( {x,y} \right)}} \\{C_{2\; R} \times {{Sobel}_{R}\left( {x,y} \right)}} & {C_{2G} \times {{Sobel}_{G}\left( {x,y} \right)}} & {C_{1B} \times {{Sobel}_{B}\left( {x,y} \right)}} \\{C_{3\; R} \times {{Sobel}_{R}\left( {x,y} \right)}} & {C_{3G} \times {{Sobel}_{G}\left( {x,y} \right)}} & {C_{1B} \times {{Sobel}_{B}\left( {x,y} \right)}}\end{pmatrix}\begin{pmatrix}{N_{R}\left( {x,y} \right)} \\{N_{G}\left( {x,y} \right)} \\{N_{B}\left( {x,y} \right)}\end{pmatrix}}} & (6)\end{matrix}$

In the equation (6), O(x,y) is a pixel value of the composite frame,N(x,y) is a pixel value of the ordinary frame, Sobel(x,y) is a pixelvalue of the post-motion correction differential filter frame, and C isa color conversion coefficient.

As is clear from the equation (6), the post-motion correction specialframe is not used in the marking and superposing compositing process.

According to the marking and superposing compositing process, since thedegree of color conversion is controlled according to the pixel valuesof the differential filter frame, the edges of the blood vessel or thelike are more strongly subjected to the color conversion, and the otherregions are not significantly subjected to the color conversion.Accordingly, it is possible to obtain a composite frame in which onlythe edges of the blood vessel or the like stand out.

The description of the image compositing process ends with the abovedescription.

As described above, according to the endoscope device 10 that serves asthe present embodiment, since the motion vectors are detected using onlythe ordinary frames and the motion correction amounts are estimatedafter correcting the detected motion vectors, it is possible toaccurately execute the motion correction of the special frame and thedifferential filter frame. Accordingly, since it is possible toaccurately align the information of the special frame of the bloodvessel, the tumor, or the like in relation to the ordinary frame, it ispossible to allow the user (a medical practitioner performing anoperation, or the like) to accurately and clearly visually recognize atumor portion to be removed and the blood vessel portion not to beremoved.

Since the composite frame that is presented to the user is created basedon the ordinary frame, a composite frame that is bright with littlenoise in comparison to the special frame can be presented to the user.

Incidentally, the series of processes described above can be executedusing hardware, and can be executed using software. When the series ofprocesses is executed using software, the program configuring thesoftware is installed on a computer. Here, examples of the computerinclude a computer embedded within dedicated hardware, and an ordinarypersonal computer or the like which is capable of executing the variousfunctions due to various programs that are installed thereon.

FIG. 9 is a block diagram illustrating a configuration example of thehardware of the computer which executes the series of processesdescribed above using a program.

In a computer 100, a central processing unit (CPU) 101, a read onlymemory (ROM) 102, and random access memory (RAM) 103 are connected toeach other by a bus 104.

Furthermore, an input-output interface 105 is connected to the bus 104.The input-output interface 105 is connected to an input unit 106, anoutput unit 107, a storage unit 108, a communication unit 109, and adrive 110.

The input unit 106 is formed of a keyboard, a mouse, a microphone, andthe like. The output unit 107 is formed of a display, a speaker, and thelike. The storage unit 108 is formed of a hard disk, non-volatilememory, or the like. The communication unit 109 is formed of a networkinterface or the like. The drive 110 drives a removable medium 111 suchas a magnetic disk, an optical disc, a magneto-optical disc, or asemiconductor memory.

In the computer 100 configured as described above, the series ofprocesses described above is performed by the CPU 101, for example,loading the program stored in the storage unit 108 into the RAM 103 viathe input-output interface 105 and the bus 104, and executing the loadedprogram.

The computer 100 may be a so-called cloud computer that is connected viathe Internet, for example.

Note that, the program which is executed by the computer 100 may be aprogram in which the processes are performed in time series in the orderdescribed in the present specification. The program may be a program inwhich the processes are performed in parallel or at the necessary timingsuch as when the process is called.

The embodiments of the present disclosure are not limited to theembodiment described above, and various modifications may be made withina scope not departing from the main concept of the present disclosure.

Furthermore, the present disclosure may adopt the followingconfigurations.

(1) An image processing device, including an input unit which inputsordinary frames in a state in which an object is irradiated withordinary light, and a special frame in a state in which the object isirradiated with special light, which are imaged consecutively at apredetermined ratio according to a predetermined frame period; adetection unit which detects motion vectors of the object from aplurality of the ordinary frames with different imaging timing; a motioncorrection unit which subjects the special frame to motion correctioncorresponding to the imaging timing of the ordinary frames based on thedetected motion vectors; and a compositing unit which subjects theordinary frames to an image compositing process based on the specialframe.

(2) The image processing device according to (1), further including afeature extraction process unit which generates a feature extractionframe by subjecting the special frame to a feature extraction process,in which the motion correction unit further subjects the featureextraction frame to motion correction corresponding to the imagingtiming of the ordinary frames based on the detected motion vectors, andin which the compositing unit subjects the ordinary frames to the imagecompositing process based on the feature extraction frame that issubjected to motion correction.

(3) The image processing device according to (2), in which the featureextraction process unit generates a differential filter frame as thefeature extraction frame by subjecting the special frame to adifferential filter process.

(4) The image processing device according to any one of (1) to (3), inwhich the compositing unit subjects the ordinary frames to a superposingcompositing process or a marking compositing process as the imagecompositing process.

(5) The image processing device according to (4), in which as thesuperposing compositing process, the compositing unit adds themotion-corrected special frame to the ordinary frames according to themotion-corrected feature extraction frame.

(6) The image processing device according to (4), in which as themarking compositing process, the compositing unit subjects the ordinaryframes to a color conversion process according to the motion-correctedfeature extraction frame.

(7) The image processing device according to any one of (4) to (6), inwhich the compositing unit subjects the ordinary frames to a superposingcompositing process or a marking compositing process as the imagecompositing process according to a selection by a user.

(8) The image processing device according to any one of (1) to (7),further including a motion vector correction unit which corrects thedetected motion vectors based on the plurality of motion vectors thatare consecutively detected.

(9) An image processing method performed by an image processing device,the method including inputting ordinary frames in a state in which anobject is irradiated with ordinary light, and a special frame in a statein which the object is irradiated with special light, which are imagedconsecutively at a predetermined ratio according to a predeterminedframe period; detecting motion vectors of the object from a plurality ofthe ordinary frames with different imaging timing; subjecting thespecial frame to motion correction corresponding to the imaging timingof the ordinary frames based on the detected motion vectors; andsubjecting the ordinary frames to an image compositing process based onthe special frame.

(10) A program for causing a computer to function as an input unit whichinputs ordinary frames in a state in which an object is irradiated withordinary light, and a special frame in a state in which the object isirradiated with special light, which are imaged consecutively at apredetermined ratio according to a predetermined frame period; adetection unit which detects motion vectors of the object from aplurality of the ordinary frames with different imaging timing; a motioncorrection unit which subjects the special frame to motion correctioncorresponding to the imaging timing of the ordinary frames based on thedetected motion vectors; and a compositing unit which subjects theordinary frames to an image compositing process based on the specialframe.

(11) An endoscope device, including a light source unit which irradiatesan object with ordinary light or special light; an imaging unit whichconsecutively images, at a predetermined ratio according to apredetermined frame period, ordinary frames in a state in which theobject is irradiated with the ordinary light, and a special frame in astate in which the object is irradiated with the special light; adetection unit which detects motion vectors of the object from aplurality of the ordinary frames with different imaging timing; a motioncorrection unit which subjects the special frame to motion correctioncorresponding to the imaging timing of the ordinary frames based on thedetected motion vectors; and a compositing unit which subjects theordinary frames to an image compositing process based on the specialframe.

It should be understood by those skilled in the art that variousmodifications, combinations, sub-combinations and alterations may occurdepending on design requirements and other factors insofar as they arewithin the scope of the appended claims or the equivalents thereof.

What is claimed is:
 1. An image processing device, comprising: an inputunit which inputs ordinary frames in a state in which an object isirradiated with ordinary light, and a special frame in a state in whichthe object is irradiated with special light, which are imagedconsecutively at a predetermined ratio according to a predeterminedframe period; a detection unit which detects motion vectors of theobject from a plurality of the ordinary frames with different imagingtiming; a motion correction unit which subjects the special frame tomotion correction corresponding to the imaging timing of the ordinaryframes based on the detected motion vectors; and a compositing unitwhich subjects the ordinary frames to an image compositing process basedon the special frame.
 2. The image processing device according to claim1, further comprising: a feature extraction process unit which generatesa feature extraction frame by subjecting the special frame to a featureextraction process, wherein the motion correction unit further subjectsthe feature extraction frame to motion correction corresponding to theimaging timing of the ordinary frames based on the detected motionvectors, and wherein the compositing unit subjects the ordinary framesto the image compositing process based on the feature extraction framethat is subjected to motion correction.
 3. The image processing deviceaccording to claim 2, wherein the feature extraction process unitgenerates a differential filter frame as the feature extraction frame bysubjecting the special frame to a differential filter process.
 4. Theimage processing device according to claim 2, wherein the compositingunit subjects the ordinary frames to a superposing compositing processor a marking compositing process as the image compositing process. 5.The image processing device according to claim 4, wherein as thesuperposing compositing process, the compositing unit adds themotion-corrected special frame to the ordinary frames according to themotion-corrected feature extraction frame.
 6. The image processingdevice according to claim 4, wherein as the marking compositing process,the compositing unit subjects the ordinary frames to a color conversionprocess according to the motion-corrected feature extraction frame. 7.The image processing device according to claim 4, wherein thecompositing unit subjects the ordinary frames to a superposingcompositing process or a marking compositing process as the imagecompositing process according to a selection by a user.
 8. The imageprocessing device according to claim 2, further comprising: a motionvector correction unit which corrects the detected motion vectors basedon the plurality of motion vectors that are consecutively detected. 9.An image processing method performed by an image processing device, themethod comprising: inputting ordinary frames in a state in which anobject is irradiated with ordinary light, and a special frame in a statein which the object is irradiated with special light, which are imagedconsecutively at a predetermined ratio according to a predeterminedframe period; detecting motion vectors of the object from a plurality ofthe ordinary frames with different imaging timing; subjecting thespecial frame to motion correction corresponding to the imaging timingof the ordinary frames based on the detected motion vectors; andsubjecting the ordinary frames to an image compositing process based onthe special frame.
 10. A program for causing a computer to function as:an input unit which inputs ordinary frames in a state in which an objectis irradiated with ordinary light, and a special frame in a state inwhich the object is irradiated with special light, which are imagedconsecutively at a predetermined ratio according to a predeterminedframe period; a detection unit which detects motion vectors of theobject from a plurality of the ordinary frames with different imagingtiming; a motion correction unit which subjects the special frame tomotion correction corresponding to the imaging timing of the ordinaryframes based on the detected motion vectors; and a compositing unitwhich subjects the ordinary frames to an image compositing process basedon the special frame.
 11. An endoscope device, comprising: a lightsource unit which irradiates an object with ordinary light or speciallight; an imaging unit which consecutively images, at a predeterminedratio according to a predetermined frame period, ordinary frames in astate in which the object is irradiated with the ordinary light, and aspecial frame in a state in which the object is irradiated with thespecial light; a detection unit which detects motion vectors of theobject from a plurality of the ordinary frames with different imagingtiming; a motion correction unit which subjects the special frame tomotion correction corresponding to the imaging timing of the ordinaryframes based on the detected motion vectors; and a compositing unitwhich subjects the ordinary frames to an image compositing process basedon the special frame.